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Background / Context: 

Description of prior research and its intellectual context. 

> Improving the quality of teacher preparation is an important national issue, because the 
quality of teaching plays such a large role in students’ learning. One key indicator of students’ 
academic success is the competence and capability of their teachers. However, the availability of 
well-prepared and effective teachers varies widely across the country, even within schools 
(Clotfelder, Ladd & Vigdor, 2007; Boyd, Grossman, Lankford & Wyckoff, 2006; Rivkin, 
Hanushek & Kain, 2005). Concerns about the highly variable quality of prospective teachers, 
coupled with teacher shortages, particularly in math and science-related fields, have generated 
both alternative pathways for entry into teaching and programs aimed at recruiting talented 
individuals to become teachers (e.g.. Teach for America, Boston Teacher Residency, New York 
City Teaching Fellows, and the New Teacher Project). 

In this presentation, we describe one component of a study that examined the impact of a 
program that awards grants to higher educational institutions to support talented individuals who 
commit to teaching mathematics or science in K-12 settings, particularly in high-need districts. 
Our presentation is focused on the design challenges and our use of a comparative short 
interrupted time series (C-SITS) approach to conducting an assessment of program impact on 
teacher certification, and entry in high-need districts. We believe that the challenges faced in the 
current study are not unique to the current study and that our approaches to dealing with the 
challenges will be informative to other educational researchers as they seek design options for 
their evaluations. 

Purpose / Objective / Research Question / Focus of Study: 

Description of the focus of the research. 

> The objectives of the teacher recruitment incentive program examined in this study are to 
recruit and train teacher candidates who are highly qualified in a STEM content area to teach in 
high-need districts. We considered two aspects of teacher recruitment - teacher certification in a 
STEM field, and teacher entry into a school located in a high-need district. The impact analyses 
on teacher outcomes therefore attempted to address the following questions: 

1. How does an IHE’s receipt of a teacher recruitment incentive grant affect its production 
of certified or licensed STEM teachers? 

2. How does an IHE’s receipt of a teacher recruitment incentive grant affect its production 
of certified or licensed STEM teachers who take teaching jobs in high-need districts? 
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Setting: 

Description of the research location. 

(May not be applicable for Methods submissions) 

> Teacher recruitment incentive grants are awarded to Institutions of Higher Education (IHEs) 
around the country. The teacher impact study is focused on several states around the country.^ 

Population / Participants / Subjects: 

Description of the participants in the study: who, how many, key features, or characteristics. 
(May not be applicable for Methods submissions) 

> The population for the study includes all IHEs in the impact states, for which we were able to 
collect sufficient data. The analytic sample is comprised of two types of IHEs. Treatment IHEs 
are those that received a teacher recruitment incentive grant. Comparison schools are IHEs with 
approved educator programs that never received a grant. 

Intervention / Program / Practice: 

Description of the intervention, program, or practice, including details of administration and 
duration. 

(May not be applicable for Methods submissions) 

> IHEs that received a teacher recruitment incentive grant provide support to STEM majors who 
enroll in their teacher preparation program and agree to work in a high-need district for two years 
for each year of support received. They receive financial support alongside other types of 
support during their teacher preparation years as well as their early years in teaching (such as 
mentoring, specialized courses, financial support to attend conferences, etc.). In return, these 
students commit to teaching in high-need schools for a specified amount of time. 

Significance / Novelty of study: 

Description of what is missing in previous work and the contribution the study makes. 

> The design challenges to the impact evaluation include the following: 

• The evaluation is of an existing program where evaluators have no control over treatment 
assignment 

• Treatment assignment (a teacher recruitment incentive grant to an IHE) occurs on a 
rolling basis, spanning many years. 

• Many of the treatment units began to receive treatment before the evaluation contract was 
awarded 

• Budgetary realities constrained the design to utilize existing administrative data sources 

• There is expected to be a lag between treatment assignment and expected impacts on 
outcome measures 

• There is often extreme year-to-year variation in outcome measures within measurement 
units (IHEs) 
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Since the findings have not been cleared for public release yet, the specific states will not be cited. 
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• Time-varying unmeasured factors unrelated to treatment can have large effects on 
outcome measures 

• Competing programs and initiatives may affect outcomes for both treatment and 
comparison units (IHEs) 

Although the same or similar challenges exist for other education evaluations, there is a paucity 
of guidance for dealing with those challenges. Our presentation is intended to provide a 
framework for understanding the challenges and to provide guidance and a base of experience on 
which to design and implement evaluations of other educational initiatives that face similar 
problems. 

Statistical, Measurement, or Econometric Model: 

Description of the proposed new methods or novel applications of existing methods. 

> We use a quasi-experimental approach to addressing the question of whether an IHE’s receipt 
of a teacher recruitment incentive grant affects its production of certified STEM teachers (or 
production of teachers that take positions in high-need schools). Our approach can be described 
as a “pre-post with comparison groups” design, a “difference-in-differences” design, or as a 
“comparative short-interrupted time series” (C-SITS) design. Our analytic models utilize 
outcome data from a time period that begins several years before any IHE in began treatment and 
continues through to the year of the most recently available data (after IHEs began receiving the 
treatment). Eike other C-SITS approaches that have been described in the literature, our model 
utilizes fixed-effects for measurement units (IHEs) and fixed effects for years. We show, 
however, that these standard models perform poorly because the year effects are not constant 
across IHEs, but rather appear to be proportional to the mean of the outcome variable in the pre- 
treatment years. In other words, IHEs with large pre-treatment means (high production of STEM 
certified teachers) have proportionately larger year effects than IHEs with smaller pre-treatment 
means. In our presentation we show how to look for a proportionate year effect and how to 
specify and fit an impact model that accounts for a proportionate year effect. In order to help our 
audience to understand the relationship between alternative models and the data, our presentation 
will feature scatter-plots overlaid with fitted values from several potential impact models. 
Because the original metric of the outcome variables (expressed as the number of STEM 
certified teacher produced per IHE per year, or the number of teachers that take jobs in high need 
schools produced per IHE per year) can vary so widely among years, we also discuss alternative 
coding of outcome variables that are less variable over time (e.g. outcomes expressed as 
proportion of all certifications produced per IHE year that are STEM certifications). 

Usefulness / Applicability of Method: 

Demonstration of the usefulness of the proposed methods using hypothetical or real data. 

> We will demonstrate our approach with illustrative data. The challenges presented in our 
evaluation are broadly applicable to other education evaluations. We expect that our 
presentation will help our audience think through the challenges and help them learn to identify 
more and less appropriate modeling strategies to meet their needs. We expect, for example that 
our audience will understand why more familiar approaches such as a more simple difference-in- 
differences specification, or a C-SITS approach with a baseline-mean projection model, a linear 
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projection model, or even a non-linear projection model would not have been good choices for 
our data, and we expect that this presentation will help the audience to make more informed 
choices in their own evaluations. 


Research Design: 

Description of the research design (e.g., qualitative case study, quasi-experimental design, 
secondary analysis, analytic essay, randomized field trial). 

(May not be applicable for Methods submissions) 

> As described above, we utilize a quasi-experimental approach that utilizes administrative data. 

Data Collection and Analysis: 

Description of the methods for collecting and analyzing data. 

(May not be applicable for Methods submissions) 

> We utilize state personnel teacher certification data and employment records. 

Findings / Results: 

Description of the main findings with specific details. 

(May not be applicable for Methods submissions) 

> In order to explain our approach prior to the report being cleared, we will present illustrative 
data and findings. 

Conclusions: 

Description of conclusions, recommendations, and limitations based on findings. 

> Our presentation will help evaluators who are considering designing a C-SITS study using 
administrative data. The presentation will help evaluators identify appropriate strategies and 
models to address their problem. It will also help them to identify limitations and pitfalls to this 
or similar approaches including the following: 

• the quality of inferences is limited by the quality of the data; 

• unlike experimental designs, causal inferences from approach described here are strongly 
dependent on assumptions regarding the correctness of the model; 

• unlike experimental designs, and some types of quasi-experimental approaches, where 
practically all modeling decisions can be specified in advance of seeing the data, full 
specification of the approach described here cannot occur before data are collected. 
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